Skip to content

Decode the 5/MG optical and raw-IMU historical layouts on Android - #2018

Merged
ryanbr merged 2 commits into
mainfrom
feat/1992-android-v20-v21
Sep 9, 2026
Merged

ryanbr merged 2 commits into
mainfrom
feat/1992-android-v20-v21

Conversation

@ryanbr

@ryanbr ryanbr commented Sep 9, 2026 •

Copy link
Copy Markdown
Owner

Was stacked on #2017, which has since merged; this now targets main and the branch was rebuilt on it,
so the diff is just the port. It still DEPENDS on what #2017 landed: the archive's decode-outcome test
now asks whether a record yielded something scoreable rather than whether it decoded, which is the
only thing that keeps a v20 record in the reject archive once it starts decoding.

Note this had NO CI while it was stacked. Every workflow is pull_request: branches: [main], so a PR
based on a branch gets zero checks rather than green ones. It is only now being verified.

What

v20 (raw optical) and v21 (raw 6-axis IMU) decoded to null on Android while Swift dispatched both. The
decoders were already here: Whoop5RawOptical and Whoop5RawImu were wired to the LIVE deep-buffer
route and never to the type-47 historical dispatch, so an offloaded record in either layout yielded
nothing at all, not even the timestamp its shared v18 header carries.

decodeWhoop5HistoricalV2021 is the port of Swift's function of the same shape. Both layouts reuse
the v18 record header, so the layout marker, monotonic record index and unix now decode. v21 emits its
six 100-sample i16 channels raw and unscaled exactly as Swift does, with the physical scales left to
Whoop5RawImu.decode, and only at a full 100 samples so a truncated record cannot produce an array
that looks complete. v20 goes through Whoop5RawOptical for block headers, per-block sample counts
and channel arrays, asserting no wavelength and no absolute unit.

What this does NOT do

It does not make those nights stageable, and is not meant to. Neither layout carries a per-second
heart rate or a gravity vector; they carry raw channels no engine reads. Mapping those channels to a
physiological value is the open work on #1992 and still needs a capture. What this buys is the same
decode on both platforms, a real timestamp and record index where there was nothing, and the arrays
available on the device.

The ordering mattered

The records must still reach the reject archive. v20 is no longer archived by the unmapped-layout rule
(it is mapped now) nor for failing to decode (it decodes now), so the only thing keeping it is the
decode-outcome test asking whether a record yielded a unix AND either a heart rate or gravity. #2017
aligned that test with Swift's first, deliberately as a no-op at the time.

Without that already in place, this commit would have silently stopped preserving the very bytes the
channel mapping needs, on the only platform collecting them. whoop5V20StillArchivedAfterItDecodes
and its v21 twin pin it, and they fail against the old filter.

Tests that flip rather than break

The mapped-set lockstep now reads {18, 20, 21, 26}, and the layout classifier answers
DECODES_WITHOUT_NAMED_SIGNAL for v20 where it answered UNMAPPED. Both were written in #2017 to
flip exactly here, so the change shows up as two repins rather than as a surprise.

Provenance

Ported from this repository's own Swift decoder. While checking it I compared against OpenStrap's Dart
decoder (MIT), which reads v20's geometry the same way: body start, block stride, slot offsets and
sign-extended 20-bit samples. No code taken from it; the agreement is corroborating evidence for our
reading, not a source for it.

Verification

  • Android suites: 5,752 tests, 0 failures. Tools/doc_comment_lint.py clean.
  • Swift untouched by this commit.
  • Not run: a real strap. A 5/MG banking v20/v21 should now log "NOOP decodes it, but these records
    carry no per-second heart rate and no motion" instead of "doesn't decode yet", and the reject archive
    should keep collecting the same records it did before.

Base automatically changed from fix/1992-frontier-and-layout-warning to main September 9, 2026 07:38
…outs

v20 (raw optical) and v21 (raw 6-axis IMU) decoded to null on Android while Swift dispatched both.
The decoders were already here: `Whoop5RawOptical` and `Whoop5RawImu` were wired to the LIVE
deep-buffer route and never to the type-47 historical dispatch, so an offloaded record in either
layout yielded nothing at all, not even the timestamp its shared v18 header carries.

`decodeWhoop5HistoricalV2021` is the port. Both layouts reuse that header, so the layout marker,
the monotonic record index and the unix now decode for records that previously produced none. v21
emits its six 100-sample i16 channels raw and unscaled, exactly as Swift does, with the physical
scales left to `Whoop5RawImu.decode`; a channel is emitted only at a full 100 samples, so a truncated
record cannot produce an array that looks complete. v20 goes through `Whoop5RawOptical` for the block
headers, per-block sample counts and channel arrays, asserting no wavelength and no absolute unit.

This does NOT make those nights stageable, and is not meant to. Neither layout carries a per-second
heart rate or a gravity vector; they carry raw channels no engine reads, and mapping those to a
physiological value is the open work on #1992. What it buys is the same decode on both platforms, a
real timestamp and record index where there was nothing, and the arrays available on the device.

The records are still archived, which is the part that had to be got right in the correct order. The
reject filter's decode-outcome test was aligned with Swift's first, so it asks whether a record
yielded a unix AND either a heart rate or gravity rather than whether it decoded at all. Without that
already in place this commit would have silently stopped preserving the very bytes the channel
mapping needs, on the only platform collecting them. `whoop5V20StillArchivedAfterItDecodes` and its
v21 twin pin exactly that, and they fail against the old filter.

Two tests flip rather than break: the mapped-set lockstep now reads {18, 20, 21, 26}, and the layout
classifier now answers DECODES_WITHOUT_NAMED_SIGNAL for v20 where it answered UNMAPPED. Both were
written to flip here.

Independently corroborated while checking this: OpenStrap's Dart decoder (MIT) reads v20's geometry
the same way we do, body start, block stride, slot offsets and sign-extended 20-bit samples. No code
taken; the agreement is evidence, not a source.

Android suites 5,752 green.
…ader

The tests shipped with the port used a short v18-derived frame with its version byte changed. On one
of those `Whoop5RawOptical.decode` refuses outright, because it requires the frame to BE a 2140-byte
buffer, so only the shared header decoded and the optical branch the port exists for was never run at
all. Every assertion passed and none of them touched it.

There is now an end-to-end case on a real-shaped, checksum-sealed 2140-byte v20 buffer: the block
count, the block's sample count, the widest-block sample total, the channel-present tally, the raw
header, and a planted i32 sample read back unmasked and unscaled through the historical dispatch. It
also asserts the record is still archived, so the guard holds on a frame that decodes fully rather
than only on one that stops at the header.

Also checked while re-reviewing, and unchanged: `decodeHistorical` now returning non-null for v20 and
v21 does NOT move the completed-offload classification. `decodedChunksThisSession` counts non-empty
StreamBatch commits, and these layouts still produce no rows, so a strap banking only them is still
classified as having banked no sensor history and still says so.

Android suites 5,753 green.
@ryanbr
ryanbr force-pushed the feat/1992-android-v20-v21 branch from 6d35fc1 to 88ee566 Compare September 9, 2026 07:42
@ryanbr
ryanbr merged commit 8c85004 into main Sep 9, 2026
3 checks passed
@ryanbr
ryanbr deleted the feat/1992-android-v20-v21 branch September 9, 2026 07:48
DX23876 added a commit to DX23876/noop that referenced this pull request Sep 16, 2026
Thirty real conflicts. The Android tree is dropped again per FORK_GUIDE; every
other hunk was resolved by asking which mechanism this fork actually runs, not
by preferring a side.

Taken from upstream:
  • Protocol: ryanbr#2017/ryanbr#2018/ryanbr#2020 — the 5/MG historical optical and raw-IMU
    layouts, and the v26 window's ABSOLUTE base code beside its deltas. Its
    migration lands as `v57-ppg-waveform-base-code` (upstream's v44), since this
    fork's sequence is nine ahead of theirs.
  • BLE diagnostics ryanbr#1998-ryanbr#2003: a bond end that says WHY, named BLE integers, a
    session-budgeted hex dump, and "we hold the link" only when the OS says so.
  • Battery: ryanbr#1951 stop asking a 5/MG for a reading it refuses; ryanbr#1959 poll
    0x2A19 for an unbonded one.
  • Scoring: ryanbr#1978/ryanbr#1982/ryanbr#1990 the RHR/HRV baseline gates, ryanbr#1977 the strain
    denominator's domain, RMSSD withheld on an over-counted night, ryanbr#1943 the
    resting-HR bin gate, the closed final HR/HRV window, HR-recovery requiring
    one CONTINUOUS effort, ryanbr#1984 nights bridged under minSleepMin, ryanbr#2015 motion
    re-folded only when that day's gravity moved, ryanbr#2016 a provisional Rest.
  • Charts ryanbr#2004/ryanbr#2005/ryanbr#2007/ryanbr#2009/ryanbr#2011 and ryanbr#1995's hero-ring detail route,
    ryanbr#1164's "Pending sync" caption under Rest.
  • Widgets: the heart-rate widget with its trace (ryanbr#1957/ryanbr#1964) and the widget
    cost counters.
  • Health: ryanbr#1503 — the Apple Health external UUID drops the strap id, so a
    re-pair no longer strands every prior record, plus the one-off sweep that
    reclaims the stranded ones; ryanbr#1205 GPS routes read back from Apple Health.
  • Storage: rolling retention for ppgWaveformSample, the skin-temp absolute
    backfill outside the 21-night window, per-table byte estimates.
  • Units split (body vs exercise distance), per-day wake overrides, the
    wind-down nudge pinned to its own day, and the German Sie→du conversion
    (29 entries, merged per LOCALE so the fork's own de edits stayed).

Kept on the fork's side, with the reason:
  • ryanbr#1005's in-memory `dayScanCache` stays out, again: the persisted
    `DayScanFingerprint` survives relaunch and its key folds the per-day weight
    and max HR upstream's does not.
  • The steps-motion cache stays PERSISTED for the same reason. Upstream's
    failed-read-vs-empty-day distinction IS adopted into it — on a persisted
    cache a stale zero from a read error would outlive the process that made it.
  • Every stream read in the day loop keeps `limit: Int.max`. All five are
    uncapped together; taking StreamReadCap for three of them would leave the
    loop half-capped, which is worse than either choice.
  • The coach engine. Upstream's K1-K15 layer is not adopted: its single 40-row
    `coachMessage` table, its tool-less pre-baked context and its own brief are
    behind what this fork already ships. Its ADDITIVE half is kept — the
    on-device voice input, the Siri "Ask Coach" intent, the brief scheduler and
    brief widget, and streaming for the OpenAI-compatible providers this fork's
    Anthropic-only StreamingToolClient does not cover. They are wired up in the
    next commit; `CoachBriefWidget` is deliberately not registered until then.
  • Liquid's `DomainTheme` labels, value-sampled Charge tint, `LiquidScoreGauge`
    and `TodayCardSurface`; classic Today's ring order and `LiveEffort.today`.
  • RootTabView's four-tab shell and `MoreCatalog`: Coach hangs on content here,
    not on a tab.

Two duplicate mechanisms collapsed rather than carried:
  • `ChargeBreakdownWiring` (upstream) and `ChargeBreakdownFormat` (fork) are the
    same extraction done twice. The fork's survives, Coupled now reads it, and
    upstream's TEST is kept — it is coverage this path never had.
  • Upstream's `CoachMessageStore` is dropped; `CoachConversationStore` already
    autosaves the whole multi-conversation history.

One defect found while resolving, fixed here: `LiveEffort.today` read HR with no
explicit limit, so `hrSamples`' 8000-row default truncated it — at ~18k rows a
day that covered about ten hours and the live Effort silently stopped climbing.
It failed safe (`effectiveEffort` takes the max), which is why it went unseen.

`Live5RestFrozenTests` is removed by upstream, which replaces it with
`Live5NoGravityRestChainTests`. `local_day_windows_oracle.json` moves out of
upstream's android path into `Tests/StrandAnalyticsTests/Oracles/` — NOT
`Fixtures/`, which this repo's .gitignore excludes everywhere.

Verified: StrandAnalytics 2233, WhoopStore 627, WhoopProtocol 700, StrandImport
293, StrandDesign 120, NoopLocalAccess 14, SemanticMemory 19, OuraProtocol 209
and PolarProtocol 25 all green; macOS and iOS both build; doc_comment_lint clean;
`i18n_audit --ci` byte-identical to its pre-merge output (it is red on main for
reasons that predate this, and the seven new widget literals were given all nine
locales so the gate did not get worse).

Six StrandTests were ALREADY failing on main before this merge and still are —
`CoachMyLogsAndZonesTests` (the tool census pin was not bumped when the Hevy
tools landed) and five in `RawPhysiologyUnionTests`. Every input to them —
CoachTools.swift, ToolConsent.swift, Repository.swift and both test files — is
byte-identical to main, so this merge neither caused nor fixed them. They went
unnoticed because app-build.yml last ran on 2026-08-25.

Analysis migration required: YES. The scoring changes above make stored derived
values stale; the recipe bump and its resumable migration are the next commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant